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WWC identified 158 
empirical studies 
of 34 middle school 
math programs 



4 studies 
met evidence 
standards 




137 studies 
did not meet 
evidence 
screens 



17 studies 
met evidence 
standards with 
reservations 



The What Works Clearinghouse (WWC) reviewed interventions 
to promote middle school students’ math knowledge and 
skillsJ Because there is some variation in how school districts 
organize middle school, we considered curricula aimed at 
students in grades 6 through 9, covering one or more of the 
following content areas: numbers and operations, algebra, 
geometry, measurement, and data analysis and probability. 

Only core, comprehensive math curricula were eligible for inclu- 
sion in this review.^ These curricula extend over the course of 
one semester or more, are central to students’ regular school 
instruction, and are based on any combination of text materi- 
als, manipulatives, computer software, videotapes, and other 
materials. 

We looked at 361 studies. Of these, 203 appeared to be stud- 
ies of practices or other interventions that did not qualify for our 
review. Of the 158 remaining studies, 21 studies of 7 curricula 
met our evidence standards, 4 without reservations and 17 with 
reservations. Altogether, the WWC looked at 34 interventions: 7 



had studies that met WWC standards with or without reserva- 
tions and 27 had studies that did not meet WWC evidence 
screens. No eligible studies were identified for an additional 16 
programs at the time of this review. (The identification of eligible 
programs ended in September 2005, and that of eligible studies 
in July 2006.) 

The WWC rated the effectiveness of middle school math 
curricula based on the available research evidence. In looking at 
math achievement for the 7 curricula: 

• / Can Learn® Pre-Algebra and Algebra had positive 
effects. 

• Saxon Middle School Math had positive effects. 

• Cognitive Tutor had potentially positive effects. 

• The Expert Mathematician had potentially positive 
effects. 

• UCSMP Algebra had potentially positive effects. ( 5 > 

Two other curricula had mixed effects on math achieve- 
ment. 



This review summarizes the second 
wave of intervention reports 
produced in 2006-07. 
www.whatworks.ed.aov 



1. Findings for math programs for the elementary school level are available In the WWC Elementary School Math Toole Report . 

2. Supplemental math programs may be considered at a later date. 
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Intervention Effectiveness Ratings for Middle School Math 

Each middle school math curriculum that had at least one study meeting WWC 
standards (with or without reservations) received a rating of effectiveness in 
math achievement. The rating aims to characterize the existing evidence, taking 
into account the quality of the research design, the statistical significance of the 
findings, the size of the difference between the average math achievement for 



students in the intervention and comparison conditions, and the consistency of 
findings across studies. 

The research evidence can be rated as positive, potentially positive, mixed, no 
discernible effects, potentially negative, or negative (see the WWC Intervention 
Rating Scheme) . Table 1 shows the effectiveness ratings for the 7 middle school 
math curricula. 



Table 1 Effectiveness ratings for 7 middle school math curricula 


Intervention name 


Rating of effectiveness 


Extent of evidence 


Coonitive Tutor® Aloebra I MD://www.cameaielearnina.com) 




Moderate to large 


Connected Mathematics Proiect (CMP) MD://connectedmath.msu.edu) 


© 


Moderate to large 


I CAN Learn® Pre-Aloebra and Aloebra (www.icanleam.com) 


o 


Moderate to large 


Saxon Middle School Math (www.saxonDubHshers.com) 


o 


Moderate to large 


The Expert Mathematician (www.exoertmath.ora) 





Small 


Transition Mathematics (httD://www.ohschool.com/atschoo!/ucsmD/index.html) 


© 


Moderate to large 


University of Chicaao School Mathematics Project (UCSMP) Aloebra 

(http://www.phschool.com/atschool/ucsmp/index.html) 


• " .-1 • " 

Vh - i-f'.-. s, , 


Moderate to large 



Note: WWC intervention reports describe each curriculum and provide information on the students, cost, and scope of use. To view the intervention reports, piease click on the program name or go to www.whatworks.ed.aov . Foilowing 
each curricuium name is the deveioper’s website address. The research evaiuated addresses some but not aii grade ievels targeted by these curricuia. Grade ieveis are reiated to student age and may affect outcomes. For a compari- 
son of targeted grade ieveis and grade ieveis in the studies reviewed by the WWC, see Appendix A2. 



O Positive effects: 
strong evidence of 
a positive effect 
with no overriding 
contrary evidence 




Potentiaiiy 
positive effects: 
evidence of a 
positive effect 
with no overriding 
contrary evidence 




Mixed effects: 
evidence of 
inconsistent effects 




No discernibie 
effects: no 
affirmative evidence 
of effects 




Potentiaiiy 
negative effects: 
evidence of a 
negative effeot 
with no overriding 
contrary evidence 



O Negative effects: 
strong evidence of 
a negative effect 
with no overriding 
contrary evidence 
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Average improvement indices 

The WWC computes an average improvement index for each study, as weii as 
an average improvement index across studies of the same intervention (see the 
Technicai Detaiis of WWC-Conducted Computations) . 

The improvement index represents the difference between the percentile rank 
of the average student in the intervention condition and the percentile rank of 
the average student in the comparison condition. It can take on values between 
-50 and +50, with positive numbers denoting results favorable to the intervention 
group. Unlike the rating of effectiveness, which is based on four factors, the 
improvement index is based only on the size of the difference between the inter- 
vention and the comparison conditions. 

Math achievement 

Math achievement includes three types of outcome measures: 

• Standardized, nationally normed achievement tests that are appropriate for 
elementary students (e.g.. Comprehensive Test of Basic Skills, Wide Range 
Achievement Test) 

• Standardized state or local tests of math achievement 

• Research-based or locally developed tests or instruments that assess 
students’ mathematical concepts or skills 

We reviewed math achievement outcomes for 7 curricula, and the average 
improvement index ranged from -2 to +14 percentile points (figure 1).® 



Figure 1 Math achievement: average improvement 




The Expert UCSMP 

Mathematician Algebra 



Cognitive Saxon Middle I CAN Learn® Transition Connected 

Tutor School Math Pre-Algebra Mathematics Mathematics 

and Algebra Project 



3. To enable comparisons across interventions, improvement indices are calculated from student-level findings. In the case of the Connected Mathematics Project (CMP) and Saxon Middle School 
Math, the average improvement index does not represent all of the findings included in the WWC intervention reports, as some findings reviewed were reported on the classroom or school level 
and student-level improvement indices could not be computed. For further details please see Technical Details of WWC-Conducted Computations . 
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Table 2 Curricula reviewed with no studies meeting WWC evidence screens 


A+ny where Learning System (no website available) 


Logo (no website avaiiable) 


Accelerated Math (http://www.renieam.com/mathrenaissance/) 


Macmillan/McGraw-Hill (http://www.mhschool.com/math/2003/studenVindex.htmi) 


Addison-Wesley Mathematics (http://www.scottforesman.com/) 


Math Advantage (http://www.hbschool.com/menus/math_advantage.htmi) 


Adventures of Jasper Woodbury Series 


Math Applications and Connections (http://www.giencoe.com/) 


(http://peabody.vanderbilt.edu/proJects/funded/Jasper/Jasperhome.html) 


Math Renaissance® (http://research. renleam. com/success/mathsuccess. asp) 


Algebra Project (http://www.algebra.org/) 


Mathematics in Context (MiC) (http://showmecenter.missouri.edu/showme/mic.shtml) 


Algebraic Thinking (http://www.aigebraicthinking.com/) 


Mathematics Plus (no website available) 


Appalachia Model Mathematics Program (no website available) 


MathScape; Seeing and Thinking Mathematically (http://www2.edc.org/mathscape/) 


CompassLearning (http://www.compassiearning.com/) 


MathThematics (http://www.mcdougallittell.com/ml/math. htm?lvl=4&ID=1005500000030872) 


Connecting Math Concepts (CMC) (https://www.sraonline.com/) 


Middle Grades Math (Scott Foresman-Addison Wesley: no website available) 


CORD Applied Math 

(http://www.cordcommunications.eom/Store/Contextual_Mathematics/CORD_Applied_Math.asp) 


Middle School Mathematics through Applications Program (MMAP) (http://mmap.wested.org/) 
Moving with Math® Extensions (http://www.movingwithmath.com/summer_math/weicome2.htm) 


Core Plus Mathematics Project (http://www.wmich.edu/cpmp/) 


Moving with Math® Math by Topic 

(http://www.movingwithmath.com/middle_school/middle_school.htm) 


Countdown Video IGAP Intervention Tape (no website available) 


Dpening Eyes to Mathematics by The Math Learning Center 

(http://www.mathleamingcenter.org/curriculum/elementary/open-eyes.asp) 


Destination Math 

(http://www.riverdeep.net/portal/page?_pageid=336,1&_dad=portal&_schema=PORTAL) 


Partnership for Access to Higher Mathematics (PATH Mathematics: no website available) 
PLATD (http://www.plato.com/) 


FUNdamentallyMATH® (http://www.fundamentallymath.com/) 


Real Math basal mathematics program (https://www.sraonline.com/rm_home.html) 


Heath Mathematics Connections (no website available) 


Reasoning Mind (http://www.reasoningmind.org/) 


Holt Middle School Math (http://go.hrw.com/gopages/ma-msm.html) 


Singapore Mathematics (http://www.singaporemath.com/) 


Integrated Mathematics, Science, and Technology (IMaST) 


The Six Through Eighth Grade Mathematics (STEM) Project (no website available) 


(http://www.cemast.ilstu.edu/programs/imasVindex.shtml) 


Scott Foresman Math Diagnostic & Intervention System 


Key Math Teach and Practice (http://ags.pearsonassessments. 


(http://www.successmaker.eom/Subscriber/1, 24/start.html) 


com/group.asp?nGrouplnfolD=a6880) 


Successmaker (http://www.pearsondigital.com/successmaker/) 


Larson Developmental Math Series (http://college.hmco.com/instmctors/catalog/demos/larson.html) 


Unitedstreaming^'” (http://www.unitedstreaming.com/) 


Lightspan Achieve Now (no website available) 



Note: Following each program name is the developer’s website address. The table includes all eligible programs with no studies and all eligible programs with no studies meeting evidence stan- 
dards. Note that some of the programs listed in this table had evaluation studies that did not meet the WWC evidence screens because the programs were supplemental curricula rather than core 
curricula. Supplemental curricula may be considered when this topic review is updated. 

For more information about studies reviewed and WWC methodoiogy, piease see the WWC Middle Schooi Math Technicai Appendices . 
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Appendix A1 Extent of evidence 



Intervention name 


Number of studies 


Sample size (schools/students) 


Extent of evidence^ 


Cognitive Tutor 


2 


9/781 


Moderate to large 


Connected Mathematics Project (CMP) 


3 


100/14,696 


Moderate to large 


1 CAN Learn® Pre-Algebra and Algebra 


6 


729/16,656 


Moderate to large 


Saxon Middle School Math 


6 


101/3,399 


Moderate to large 


The Expert Mathematician 


1 


1/70 


Small 


Transition Mathematics 


3 


49/972 


Moderate to large 


UCSMP Algebra 


2 


4/225 


Moderate to large^ 



nr = not reported 

1. A rating of “moderate to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 

2. The extent of evidence for UCSMP Algebra is considered to be moderate to large because, across studies, 14 classrooms were included at the time of analysis. 



WWC Topic Report Middle School Math 



July 30, 2007 



5 







Appendix A2 Targeted population 



Intervention name 


Targeted students 
(grade levels) 


Students in studies reviewed 
(grade levels)^ 


Cognitive Tutor 


7-12 


9 


Connected Mathematics Project (CMP) 


6-8 


6-8 


I CAN Learn® Pre-Algebra and Algebra 


6-12 


8-9 


Saxon Middle School Math 


6-9 


6-9 


The Expert Mathematician 


6-9 


8 


Transition Mathematics 


7-12 


7-9 


UCSMP Algebra 


7-10 


8-9 



Note: This table compares targeted grade levels and the grade levels in the studies reviewed by the WWC. Grade levels are related to student age and may affect outcomes due to differences in the 
students’ developmental stages as well as differences in school size and organization. 

1. Some of the studies reviewed included students in grades 10 or above, but the findings for those students were not reviewed because those grades were outside the scope of this review. 
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Appendix A3 Summary of statistically significant^ or substantively important^ positive outcomes 





Math achievement 


Intervention name 


Statistically significant positive findings^ 


Math achievement across outcomes 


Cognitive Tutor 

Morgan & Ritter, 2002 


Math achievement grades (end of first and second semesters) 


Statistically significant. Substantively important 


Schneyderman, 2001 


ns 


ns, nsi 


Connected Mathematics Project (CMP) 

Ridgway, Zawojewski, Hoover, & Lambdin, 2002 


ns 


ns, nsi 


Riordan & Noyce, 2001 


Massachusetts Comprehensive Assessment 
System (MCAS) — math scores 


US'* 


Schneider, 2000 


ns 


ns, nsi 


/ CAN Learif Pre-Atgebra and Atgebra 

Kirby, 2006, October 


Louisiana Educational Assessment Program 
(LEAP) Grade 8 Mathematics Exam 


Statistically significant. Substantively important 


Kerstyn, 2001, Algebra 1 


ns 


ns, nsi 


Kerstyn, 2001, Algebra 1 Honors 


ns 


ns, nsi 


Kerstyn, 2001, MJ-3 pre-algebra 


ns 


ns, nsi 


Kerstyn, 2001, MJ-3 Advanced 


ns 


ns, nsi 


Kerstyn, 2002, October, Algebra 1 


ns 


ns, nsi 


Kerstyn, 2002, October, Algebra 1 Honors 


ns 


ns, nsi 


Kerstyn, 2002, October, MJ-3 pre-algebra 


FCAT mathematics 


Statistically significant, nsi 


Kerstyn, 2002, October, MJ-3 Advanced 


ns 


ns, nsi 


Kirby, 2004, September 


General Mathematics CST 


Statistically significant. Substantively important 


Kirby, 2004a, November 


Georgia Criterion-Referenced Competency Test (GCRCT) Math Test 


Statistically significant. Substantively important 


Kirby, 2005, January 


Algebra 1 EOC test 


Statistically significant. Substantively important 


Saxon Middte Schoot Math 






Williams, 1986 


End-of-course math test 


Statistically significant. Substantively important 


Peters, 1992 


ns 


ns, nsi 


Crawford & Raia, 1986 


The California Achievement Test (CAT) 


Statistically significant. Substantively important 


Resendez, Fahmy, & Manley, 2005 


The Texas Assessment of Academic Skills (TAAS) — TLI score; 
The Texas Assessment of Knowledge and Skills (TAKS) 


Statistically signiticant, nsi 


Resendez & Manley, 2005 


ns 


US'* 


Roberts, 1994 


ns 


ns, nsi 


(continued/ 
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Appendix A3 Summary of statistically significant^ or substantively important^ positive outcomes (continued) 





Math achievement 




Intervention name 


Statistically significant positive findings^ 


Math achievement across outcomes 


The Expert Mathematician 

Baker, 1997 


ns 


ns, Substantively important 


Transition Mathematics 

Baker, 1997 


ns 


ns, Substantiveiy important negative effect 


Hedges et al., 1986 


Geometry Readiness 


ns, nsi 


Thompson etal., 2005 


ns 


ns, nsi 


UCSMPAigebra 

Peters, 1992 


ns 


ns, nsi 


Thompson etal., 2006 


Algebra Readiness; Problem Solving and Understanding 


ns, Substantively important 



na = not studied 

ns = not statistically significant 

nsi = not substantively important 



1 . According to the WWC criteria, if a program finds a stafistically significanf effect, there is less than a 5% chance that this difference is due fo chance. The level of stafistical significance was calculafed by fhe WWC and, where necessary, 
corrects for clusfering wifhin classrooms or schools, and for mulfiple comparisons. The level of sfafisfical significance was reporfed by the study authors or, where necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for mulfiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See the Technical Details of WWC-Conducted Computations for the formulas fhe WWC used fo calculate the statistical significance. 

2. For rafing purposes, fhe WWC considers the statistical significance of the findings and fhe magnitude of the effect, also called the effect size. An average effect size is the sum of all fhe effect sizes of the student outcomes in a study in a 
single domain divided by the number of fhose oufcomes. The WWC considers an average effect size across all student outcomes in one study in a given domain to be substantively important if if is equal fo or greater than 0.25. 

3. No studies showed statistically significant negative effects on math achievement. 

4. Student-level effect size could not be computed for this study; whether or not the magnitude of fhe effecf is subsfanfively importanf is unknown. However, the statistical significance for fhis sfudy is comparable fo ofher sfudies and is 
included in fhe intervenfion rafing. For furfher defails, see Technical Defails of WWC-Conducted Comoutations . 
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Appendix A4 
Methodology 



WWC Topic Report 



One hundred and fifty-eight studies provided data on 34 middle 
schooi math curricula and were classified by the strength of their 
designs.^ To be fully reviewed, a study had to be a randomized 
controlled trial or a quasi experimental design with evidence of 
equating between the treatment and comparison groups. 

Eligibility for review 

Quasi experiments eligible for review include those equating 
through matching or statistical adjustment, regression disconti- 
nuity designs, and single case designs. However, no studies 
identified for the middle school math review used regression 
discontinuity or single case designs. 

In judging the quality of the evidence, the review considered 
the properties of measurement instruments used in the studies, 
the percentage of the original study sample that was lost to 
follow-up, and any sample characteristics or events that might 
serve as alternative explanations for the observed effect. For 
details please see the WWC Evidence Standards . When results 
were reported for multiple time periods following sample enroll- 
ment, the longer term results were included in the review. 

The research evidence for programs that have at least 
one study meeting WWC evidence standards with or without 
reservations is summarized in individual intervention reports 
posted on the WWC website. See http://www.whatworks. 
ed.gov . So far, 21 studies of 7 middle school programs have met 
evidence standards with or without reservations. The lack of 
evidence for the remaining programs does not mean that those 
programs are ineffective; some programs have not yet been 
studied using a study design that permits the WWC to draw any 
conclusions about their effectiveness. And some studies were 
not considered for rating of effectiveness purposes because 
insufficient information was reported to enable us to confirm 
statistical findings. 



Rating of effectiveness 

Each middle school math curriculum that had at least one study 
meeting WWC standards with or without reservations received 
a rating of effectiveness for math achievement. The rating of 
effectiveness aims to characterize the existing evidence base on 
the intervention within a given domain. The intervention effects 
based on the research evidence are rated as positive, potentially 
positive, mixed, no discernible effects, potentially negative, or 
negative. 

The rating of effectiveness takes into account four factors: 
the quality of the research design, the statistical significance of 
the findings, the size of the difference between students in the 
intervention and the comparison conditions, and the consistency 
in findings across studies (see the WWC Intervention Rating 
Schemed . 

The level of statistical significance was reported by the study 
authors or, where necessary, calculated by the WWC to correct 
for clustering within classrooms or schools and for multiple com- 
parisons. Because of these corrections, the level of statistical 
significance as calculated by the WWC may differ from the one 
originally reported by the study authors. For the formulas that we 
used to calculate statistical significance, see Technical Details 
of WWC-Conducted Computations . For an explanation, see the 
WWC Tutorial on Mismatch . If the average effect size across all 
outcomes in one study in a single domain is at least 0.25, it is 
considered substantively important, contributing toward the rat- 
ing of effectiveness. See the technical appendices of the middle 
school math intervention report for further details. 

Extent of evidence 

The evidence base rating represents the size and number of 
independent samples that were assessed for the purposes of 
analysis of the program effects. A “moderate to large” evidence 



1. No empirical studies were identified for additionai 14 programs during the time period of this review. 
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Appendix A4 
Methodology 

(continued) 



WWC Topic Report 



base requires at least two studies and two schools across stud- 
ies of at least 350 students or 14 classrooms. Otherwise, the 
evidence base is considered to be “small.” The WWC Is currently 
working to define a “large” evidence base. This term should not 
be confused with external validity, as other facets of external 
validity— such as variations in settings, important subgroups of 
students, implementation, and outcome measures— were not 
taken into account for the purposes of this rating. 

Improvement Index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 



computes an average improvement index for each domain and 
each study as well as a domain average improvement index 
across studies of the same intervention (see the Technical 
Details of WWC Conducted Computations) . The improvement 
index represents the difference between the percentile rank of 
the average student in the intervention condition and the percen- 
tile rank of the average student in the comparison condition. The 
improvement index can take on values between -50 and +50, 
with positive numbers denoting results favorable to the interven- 
tion group. Unlike the rating of effectiveness, the improvement 
index is based only on the size of the difference between the 
intervention and the comparison conditions. 
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1 . Confound: there was only one intervention unit and/or one comparison unit, so the analysis could not separate the effects of the intervention from other factors. 

2. Lack of evidence for baseline equivalence: the study, which uses a quasi-experimental design, does not establish that the comparison group was 
equivalent to the intervention group at baseline. 

3. Intervention is not relevant: the intervention does not meet the WWC standards of a core middle school math curriculum. 

4. Study is outside the time frame of the review: the parameters for this WWC review specified that interventions were implemented after 1983 but this 
study involves students that began the intervention prior to 1983. 

5. Intervention is not relevant: the implementation length of the curriculum is too short. 

6. Does not use a strong causal design: this study does not use a comparison group. 

7. Does not use a strong causal design: this study provides no information on the research design and has no authorship. 

8. Outcomes measures are not relevant to this review. 

9. Does not use a strong causal design: this is a qualitative study. 

10. Lack of evidence for baseline equivalence: the study, which was reviewed as a quasi-experimental design, does not establish that the comparison group 
was equivalent to the intervention group at baseline. This study, which was designed as a regression discontinuity design, does not properly assign 
students at the cutoff grade. 

11. Does not use a strong causal design: there was a change in instrumentation during the study. 

12. Sample is not relevant to this review: the parameters for this WWC review specified that students should be in grades 6-9; this study did not disag- 

gregate students in the eligible range from those outside the range. 

13. Complete data were not reported: the WWC could not compute effect sizes. 

14. Sample is not relevant to the scope of this review: this study does not focus on students in U.S. schools, one of the parameters for this WWC review. 
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